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SUMMARY 


Problem 


For  several  years,  the  Naval  Academy  has  used  widely-administered  commercial 
measures  of  scholastic  aptitude  to  assess  academic  potential.  College  applicants  may 
take  such  tests  as  frequently  as  they  desire  and  must  submit  all  scores.  The  Naval 
Academy  uses  only  the  highest  score  for  selection.  This  strategy  may  introduce 
prediction  errors. 

Objective 

The  objective  of  this  effort  was  to  compare  the  relative  validity  of  the  average 
college  aptitude  test  score  (AVECATS)  and  the  highest  college  aptitude  test  score 
(HIGHCATS)  in  predicting  academic  success  at  the  Naval  Academy. 

Approach 

For  each  subject,  scores  were  examined  to  identify  both  the  HIGHCATS  and 
AVECATS.  The  college  aptitude  tests  included  the  verbal  and  math  subtests  of  the 
Scholastic  Aptitude  Test  (SAT)  and/or  the  American  College  Test  (ACT). 

Validities  for  predicting  Naval  Academy  academic  performance  were  computed  for 
HIGHCATS  and  AVECATS.  The  measure  of  performance  was  midshipman's  academic 
quality  point  ratio  (AQPR)  at  the  end  of  plebe  year.  Validities,  midshipmen  rank  order, 
and  predicted  AQPR  were  compared  between  HIGHCATS  and  AVECATS. 

Results 


1.  There  were  large  changes  in  position  when  midshipmen  were  rank  ordered  on  the 
AVECATS  instead  of  on  the  HIGHCATS. 

2.  For  the  total  score,  verbal  subtest,  and  math  subtest,  the  validity  of  the 
AVECATS  was  significantly  higher  than  that  of  the  HIGHCATS. 

3.  More  applicants  were  predicted  to  attain  a  higher  AQPR  when  the  AVECATS  was 
used  as  the  selection  score  th.ji  when  HIGHCATS  was  used. 

Conclusions  \ 


1.  The  AVECATS  is  a  better  predictor  of  Naval  Academy  performance  than  is  the 
HIGHCATS. 

2.  A  better  qualified  group  of  applicants  would  be  selected  for  admission  to  the 
Naval  Academy  if  AVECATS  were  used  as  the  selection  score. 

Recommendation 


Applicants  to  the  Naval  Academy  should  be  selected  based  on  their  AVECATS  instead 
of  their  HIGHCATS. 
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INTRODUCTION 


Problem  and  Background 

To  select  the  applicants  to  the  Naval  Academy  most  likely  to  succeed  in  the  officer 
training  program,  there  is  a  continuing  need  to  improve  selection  procedures.  For  several 
years,  the  academy  has  used  widely-administered  commercial  measures  of  scholastic 
aptitude  to  assess  academic  potential.  College  applicants  may  take  such  tests  as 
frequently  as  they  desire  and  must  submit  all  scores.  The  Naval  Academy  uses  only  the 
highest  score  for  selection.  This  strategy  may  introduce  prediction  errors. 

It  has  been  postulated  that  the  average  score  is  the  best  indicator  of  a  group  of 
scores  that  measure  the  same  ability  (Ferguson,  1966).  This  theory  is  supported  by 
Gulliksen  (1965)  in  his  discussion  of  doubling  the  length  of  a  test.  He  demonstrated  that 
adding  a  parallel  test  to  the  original  test  increased  its  reliability  and  validity. 

Objective 

The  objective  of  this  effort  was  to  compare  the  relative  validity  of  the  average 
college  aptitude  test  score  (AVECATS)  to  the  highest  college  aptitude  test  score 
(HIGHCATS)  in  the  prediction  of  academic  success  at  the  Naval  Academy. 


APPROACH 


Sample 

The  applicant  sample  included  6883  applicants  who  submitted  more  than  one  college 
aptitude  test  score  to  the  Naval  Academy  (class  of  1983). 

The  midshipman  sample  included  588  midshipmen  of  the  Naval  Academy  class  of  1983 
who  met  the  following  criteria: 

1.  Their  HIGHCATS  was  available  from  the  Educational  Testing  Service  and/or  the 
American  College  Testing  Program. 

2.  They  had  received  a  final  plebe  academic  quality  point  ratio. 

3.  They  had  submitted  more  than  one  college  aptitude  test  score. 


Measures 


The  predictors  included  the  scores  on  the  verbal  and  math  subtests  of  the  Scholastic 
Aptitude  Test  (SAT)  and  the  American  College  Test  (ACT). 

The  necessary  college  aptitude  test  score  data  were  available  only  for  the  class  of 
1983.  At  the  time  of  data  analysis,  academic  performance  for  this  class  was  limited  to 
plebe  year.  However,  results  obtained  from  earlier  Naval  Academy  classes,  based  on 
academic  performance  for  all  4  years,  demonstrated  that  college  aptitude  test  score 
validities  for  plebe  academic  performance  did  not  differ  significantly  from  those  for  the 
third,  second,  and  first  class  years.1  Further,  a  comprehensive  review  of  validity  studies 


Validities  found  for  class  of  1981  were  .43  for  plebe  year,  .46  for  third  class  year, 
.44  for  second  class  year,  and  .44  for  first  class  year. 
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among  civilian  colleges  revealed  no  systematic  differences  between  the  validities  of  the 
4-year  college  cumulative  grade  point  averages  and  those  of  the  first-year  college  grade 
point  averages  (Angoff,  1971).  Consequently,  the  criterion  was  the  midshipman's  grade 
point  average  at  the  end  of  plebe  year,  referred  to  as  the  academic  quality  point  ratio 
(AQPR). 

Analyses 


1.  Comparison  between  SAT  and  ACT  scores.  Initially,  separate  analyses  were 
performed  for  the  SAT  and  the  ACT  scores.  However,  results  from  both  analyses  were 
similar  and  both  the  SAT  and  the  ACT  scores  were  found  to  be  reliable  and  valid  (Angoff, 
1971;  Hills,  1979).  Therefore,  ACT  scores  were  converted  to  SAT  equivalencies  (United 
States  Naval  Academy,  1978). 

2.  Comparison  between  HIGHCATS  and  AVECATS.  The  difference  between  each 
midshipman's  HIGHCATS  and  AVECATS  was  examined.  Further,  each  midshipman  was 
ranked  within  the  sample  on  his  or  her  HIGHCATS  and,  subsequently,  on  AVECATS;  the 
difference  in  rank  order  position  for  each  midshipman  was  examined. 

3.  Comparison  of  validities.  On  the  midshipman  sample,  the  validities  of  the 
HIGHCATS  and  AVECATS  were  compared  on  the  total,  verbal  subtest,  and  math  subtest 
scores.  Further,  all  midshipmen  were  classified  into  the  following  subgroups  on  both  their 
HIGHCATS  and  AVECATS:  upper  20  percent,  upper  40  percent,  upper  60  percent,  upper 
80  percent,  and  total  group.  The  mean  AQPRs  of  the  HIGHCATS  subgroups  were 
compared  to  the  mean  AQPRs  of  the  AVECATS  subgroups. 

4.  Comparison  of  applicant  predicted  scores.  For  each  applicant,  predicted  AQPR 
was  derived  for  their  HIGHCATS  and  AVECATS.  For  both  HIGHCATS  and  AVECATS,  the 
number  of  applicants  selected  at  various  predicted  AQPR  cutoff  points  was  examined. 


RESULTS  AND  DISCUSSION 

Differences  Between  HIGHCATS  and  AVECATS 

The  distribution  of  HIGHCATS  and  AVECATS  (total  score)  for  the  midshipmen  is 
shown  in  Table  1.  AVECATS  tended  to  be  lower  than  HIGHCATS.  For  instance,  277 
midshipmen  scored  1250  and  above  in  the  HIGHCATS  distribution,  while  only  122 
midshipmen  scored  1250  and  above  in  the  AVECATS  distribution.  Table  2,  which  presents 
differences  between  HIGHCATS  and  AVECATS  (verbal  and  math  subtest  scores),  indicates 
a  nonconstant  drop  between  HIGHCATS  and  AVECATS.  Some  midshipmen  dropped  as 
little  as  10  points,  while  others  dropped  more  than  90.  Changes  in  rank  order  position 
between  HIGHCATS  and  AVECATS  are  shown  in  Table  3.  There  is  a  wide  distribution  of 
number  of  positions  change  with  an  average  change  of  about  64  positions. 

Differences  Between  Validities 


A  comparison  of  validities  by  total  test,  verbal  subtest,  and  math  subtest  for  the 
midshipmen  sample  is  shown  in  Table  4.  The  validity  of  the  AVECATS  was  significantly 
higher  than  that  of  the  HIGHCATS  for  both  subtests  and  for  the  total  score.  For  the 
verbal  subtest,  the  AVECATS  accounted  for  nearly  twice  the  variance  of  the  HIGHCATS 
in  the  criterion;  for  the  math  subtest,  the  AVECATS  accounted  for  62  percent  more 
variance;  for  the  total  test,  the  AVECATS  accounted  for  60  percent  more  variance. 


Table  1 


Distribution  of  Highest  College  Aptitude  Test  Scores  (HIGHCATs) 
and  Average  College  Aptitude  Test  Scores  (AVECATs)  (N  =  588) 


HIGHCATS 

AVECATS 

Number  of 

Number  of 

Score 

Midshipmen 

Percentage 

Midshipmen 

Percentage 

Less  than  950 

3 

0.5 

9 

1.5 

950  to  1049 

13 

2.2 

62 

10.5 

1050  to  1149 

92 

15.6 

180 

30.6 

1150  to  1249 

203 

34.5 

215 

36.6 

1250  to  1349 

187 

31.8 

98 

16.7 

1350  to  1449 

76 

12.9 

21 

3.6 

More  than  1450 

14 

2.4 

3 

0.5 

Total 

588 

99.9 

588 

100.0 

Table  2 

Differences  Between  HIGHCATS  and  AVECATS  (N  =  588) 

Verbal  Subtest 

Math  Subtest 

Difference 

(Number  of 

(Number  of 

(in  points) 

Midshipmen) 

Midshipmen) 

Less  than  10 

73 

57 

10  to  19 

75 

104 

20  to  29 

115 

116 

30  to  39 

95 

119 

40  to  49 

56 

68 

50  to  59 

53 

40 

60  to  69 

35 

33 

70  to  79 

26 

20 

80  to  89 

26 

10 

More  than  89 

34 

21 

Total 

588 

588 

3 


Table  3 


Changes  in  Rank  Order  Position  Between  HIGHCATS 
and  AVECATS  (N  =  588) 


Positions  Changed 

Number  of 

Midshipmen 

Percentage 

Less  than  21 

156 

26.5 

21  to  40 

115 

19.6 

41  to  60 

86 

14.6 

61  to  80 

80 

13.6 

81  to  100 

59 

10.0 

101  to  200 

82 

13.9 

More  than  200 

10 

1.7 

Total 

588 

99.9 

Table  4 

Comparison  of  Validities  Between  HIGHCATS 
and  AVECATS  (N  =  588) 


Test  and  Score 

Mean 

SD 

r 

r2 

Verbal  Subtest 

HIGHCATS 

578 

68 

.261 

.068 

AVECATS 

539 

66 

.342* 

.117 

Math  Subtest 

HIGHCATS 

662 

61 

.345 

.119 

AVECATS 

627 

62 

.442* 

.195 

Total  Score 

HIGHCATS 

1240 

103 

.376 

.141 

AVECATS 

1167 

105 

.479* 

.229 

*p  <  .001. 


Mean  AQPR 


Mean  AQPRs  of  HIGHCATS  and  AVECATS  subgroups  are  plotted  by  college  aptitude 
test  score  group  in  Figure  1.  Except  for  one  subgroup  (total),  all  AVECATS  subgroups 
attained  higher  mean  AQPRs  than  did  the  corresponding  HIGHCATS  subgroups. 


College  Aptitude  Test  Score  Groups 


Figure  1.  Mean  AQPR  by  college  aptitude  test  score  group. 
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Deferences  Among  Predicted  AQPR 


Figure  2  presents  the  number  of  applicants  that  would  be  selected  at  various  AQPR 
cutoff  points  for  both  HIGHCATS  and  AVECATS.  A  higher  AQPR  was  predicted  for  more 
applicants  when  the  AVECATS  was  used  as  the  selection  score  than  when  the  HIGHCATS 
was  used.  For  example,  an  AQPR  of  2.7  or  more  was  predicted  for  1074  applicants  using 
AVECATS  as  the  predictor,  but  only  for  873  applicants  using  HIGHCATS. 


Figure  2.  Number  of  applicants  selected  at  various  predicted  AQPR  cutoff  points  (N  =  6882). 
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CONCLUSIONS 


1.  The  AVECATS  is  a  better  predictor  of  NavaJ  Academy  performance  than  is  the 
HIGHCATS. 

2.  A  better  qualified  group  of  applicants  would  be  selected  for  admission  to  the 
Naval  Academy  if  AVECATS  were  used  as  the  selection  score. 


RECOMMENDATION 

Applicants  to  the  Naval  Academy  should  be  selected  on  the  basis  of  their  AVECATS 
instead  of  their  HIGHCATS. 
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